The Internet contains many types of downloadable media content items, including audio, video, documents, and so forth. These content items are often very large, such as video in the hundreds of megabytes. Users often retrieve documents over the Internet using Hypertext Transfer Protocol (HTTP) through a web browser. One type of large data includes very large images. Although many computer users are familiar with relatively small images taken by a digital camera, large images are becoming more useful. For example, websites such as MICROSOFT VIRTUAL EARTH and TerraServer pioneered making satellite imagery of the Earth available to web users. Because users can zoom anywhere from the whole planet down to a single house, this imagery can include thousands of separate tiled images as well as multiple zoom levels and resolutions. Even digital camera images are becoming larger (e.g., 10-20 megapixel) than what can easily be viewed on a computer screen all at once.
Large images are frequently stored in a tiled format, with each part of the image stored in a separate file. Deploying millions of files for a large set of large images is too slow and in some cases prevents projects from ever completing. Previous solutions store each part of the image in a separate file. Deploying these files to the public server may take months for certain large projects. Project administrators have trouble managing the thousands or millions of files created when generating such content. Moving them around from disk to disk can be very time consuming. Even worse, when it is time to publish the content to a Content Delivery Network (CDN), such as Limelight or Akamai, the CDNs use File Transfer Protocol (FTP) to copy the files. FTP is incredibly slow for dealing with many small files.
Several solutions have been tried for handling these problems, and have failed or have significant drawbacks. One solution utilizes HTTP byte range requests to retrieve an appropriate portion of content from a binary container of the many image files. Unfortunately, CDNs do not optimize byte range requests. Another solution is to zip files (i.e., compress many files into a single package file) for transfer from an origin to the server, but many CDNs do not allow sufficient access to their servers to unzip the content at the destination so that the content can be accessed by clients. There are server modules that allow leaving the content as a zip file and accessing the relevant portions upon request, but many CDNs will not install such modules on their servers. In addition, since the ZIP container indexes by string instead of integer, there is a small amount of extra overhead. Another solution leverages the CDN's custom origin model and hosts the content on the publisher's server, with the CDN acting as a pure cache. The previous solutions can then be used on the publisher's server. However, this is not popular with CDN customers because it means the publisher's server has to be fully reliable. CDN customers would usually prefer to push content to the CDN and be done.